故障分析 | 如何提高 MHA 的网络容忍能力？（下）
爱可生交付服务部团队北京 DBA，主要负责处理 MySQL 的 troubleshooting 和我司自研数据库自动化管理平台 DMP 的日常运维问题，对数据库及周边技术有浓厚的学习兴趣，喜欢看书，追求技术。
说明
本节主要介绍对 secondary_check_script 调用 masterha_secondary_check 脚本的说明。
[root@yang-03 ~]# cat /usr/bin/masterha_secondary_check
#!/usr/bin/env perl
···
$| = 1;
GetOptions(
'help' => \$help,
'version' => \$version,
'secondary_host=s' => \@monitoring_servers,
'user=s' => \$ssh_user,
'port=s' => \$ssh_port,
'options=s' => \$ssh_options,
'master_host=s' => \$master_host,
'master_ip=s' => \$master_ip,
'master_port=i' => \$master_port,
'master_user=s' => \$master_user,
'master_password=s' => \$master_password,
'ping_type=s' => \$ping_type,
'timeout=i' => \$timeout,
);
if ($version) {
print "masterha_secondary_check version $MHA::ManagerConst::VERSION.\n";
exit 0;
}
if ($help) {
pod2usage(0);
}
unless ($master_host) {
pod2usage(1);
}
sub exit_by_signal {
exit 1;
}
local $SIG{INT} = $SIG{HUP} = $SIG{QUIT} = $SIG{TERM} = \&exit_by_signal;
$ssh_user = "root" unless ($ssh_user);
$ssh_port = 22 unless ($ssh_port);
$master_port = 3306 unless ($master_port);
if ($ssh_options) {
$MHA::ManagerConst::SSH_OPT_CHECK = $ssh_options;
}
$MHA::ManagerConst::SSH_OPT_CHECK =~ s/VAR_CONNECT_TIMEOUT/$timeout/;
# 0: master is not reachable from all monotoring servers
# 1: unknown errors
# 2: at least one of monitoring servers is not reachable from this script
# 3: master is reachable from at least one of monitoring servers
my $exit_code = 0;
foreach my $monitoring_server (@monitoring_servers) {
my $ssh_user_host = $ssh_user . '@' . $monitoring_server;
my $command =
"ssh $MHA::ManagerConst::SSH_OPT_CHECK -p $ssh_port $ssh_user_host \"perl -e "
. "\\\"use IO::Socket::INET; my \\\\\\\$sock = IO::Socket::INET->new"
. "(PeerAddr => \\\\\\\"$master_host\\\\\\\", PeerPort=> $master_port, "
. "Proto =>'tcp', Timeout => $timeout); if(\\\\\\\$sock) { close(\\\\\\\$sock); "
. "exit 3; } exit 0;\\\" \"";
my $ret = system($command);
$ret = $ret >> 8;
if ( $ret == 0 ) {
print
"Monitoring server $monitoring_server is reachable, Master is not reachable from $monitoring_server. OK.\n";
next;
}
if ( $ret == 3 ) {
if ( defined $ping_type
&& $ping_type eq $MHA::ManagerConst::PING_TYPE_INSERT )
{
my $ret_insert;
my $command_insert =
"ssh $MHA::ManagerConst::SSH_OPT_CHECK -p $ssh_port $ssh_user_host \'"
. "/usr/bin/mysql -u$master_user -p$master_password -h$master_host "
. "-e \"CREATE DATABASE IF NOT EXISTS infra; "
. "CREATE TABLE IF NOT EXISTS infra.chk_masterha (\\`key\\` tinyint NOT NULL primary key,\\`val\\` int(10) unsigned NOT NULL DEFAULT '0'\) engine=InnoDB; "
. "INSERT INTO infra.chk_masterha values (1,unix_timestamp()) ON DUPLICATE KEY UPDATE val=unix_timestamp()\"\'";
my $sigalrm_timeout = 3;
eval {
local $SIG{ALRM} = sub {
die "timeout.\n";
};
alarm $sigalrm_timeout;
$ret_insert = system($command_insert);
$ret_insert = $ret_insert >> 8;
alarm 0;
};
if ( $@ || $ret_insert != 0 ) {
print
"Monitoring server $monitoring_server is reachable, Master is not writable from $monitoring_server. OK.\n";
next;
}
}
print "Master is reachable from $monitoring_server!\n";
$exit_code = 3;
last;
}
else {
print "Monitoring server $monitoring_server is NOT reachable!\n";
$exit_code = 2;
last;
}
}
exit $exit_code;
···
看到上面的脚本有点头大，作为一个专一的 DBA，一直追求着 shell，Perl 语言不熟啊，细看之后发现与 shell 脚本有异曲同工之妙，从相应的注释中，我们找到了该检查脚本的返回值，即 0,1,2,3，分别代表的含义如下：
# 0: master is not reachable from all monotoring servers
# 1: unknown errors
# 2: at least one of monitoring servers is not reachable from this script
# 3: master is reachable from at least one of monitoring servers
下面就让我们做几个测试体会体会吧。
验证
本实验使用 iptables 进行网络故障场景的模拟，以下图中四条链路是否连通来进行测试，查看对应监控脚本的返回值是否和上面讲的一致。
检查命令：
shell > masterha_secondary_check -s remote_host1 -s remote_host2 .. --user=root --master_host=master_ip --master_ip=master_ip --master_port=3306 --master_user=mha --master_password=*** --ping_type=SELECT
test 1
output：
Master is reachable from remote_host1!
test 2
output：
Master is reachable from remote_host1!
test 3
output：
ssh: connect to host remote_host1 port 22: Connection timed out.
Monitoring server remote_host1 is NOT reachable!
test 4
output：
Master is reachable from remote_host1!
test 5
output：
Monitoring server remote_host1 is reachable, Master is not reachable from remote_host1. OK.
Master is reachable from remote_host2!
test 6
output：
ssh: connect to host remote_host1 port 22: Connection timed out.
Monitoring server remote_host1 is NOT reachable!
test 7
output：
当输入的命令有误时，即符合 unknown errors，返回值为 1。
test 8
output：
Monitoring server remote_host1 is reachable, Master is not reachable from remote_host1. OK.
Monitoring server remote_host2 is reachable, Master is not reachable from remote_host2. OK.
PS：这里有一个问题，在使用监控服务器 slave 检查主从之间网络连通性时，masterha_secondary_check 脚本对监控主机的使用存在优先级，它会优先使用 remote_host1，如果 remote_host1 与 master 之间网络不同，才会使用 remote_host2 检查与 master 之间的网络连通性。
总结
通过测试验证，发现 MHA 通过调用 secondary_check_script 参数对于网络问题具有一定的容忍能力，只是相关文档描述过于简略，如需继续使用，需要不断进行验证测试，总体来讲，MHA 的功能还是及其强大的。
相关推荐：
社区近期动态